Partial Reinforcement Extinction Effect Research Articles

Inhibitory S-R Associations Influence the Partial Reinforcement Extinction Effect

The Partial Reinforcement Extinction Effect (PREE) is the finding of more rapid extinction to a continuously reinforced cue than to a partially reinforced cue. Despite its well-documented existence, it has challenged many theoretical accounts of learning that assume the likelihood of responding is correlated with the strength of the learned Stimulus (S) – Response (R) association. While a number of explanations (notably by Amsel and Capaldi) of the PREE have been proposed, Rescorla (1999) reported a PREE using an entirely within-subjects design—a finding that runs counter to these proposed explanations. We describe how the PREE might be explained by the formation of inhibitory S-R associations that develop during the acquisition of partial reinforcement, but not continuous reinforcement. To test this proposal, we trained pigeons to peck a response key in the presence of one partially reinforced stimulus but also omitted the response key during the training of an alternative partially reinforced stimulus—thus preventing the development of inhibitory S-Rs to that latter stimulus. We found evidence for the PREE only for the partially reinforced stimulus where inhibitory S-Rs were capable of being formed, thus supporting our proposal. Alternative explanations of our procedure and its relevance to broader computational rules of learning is discussed.

Learning about trial sequences disrupts the partial reinforcement extinction effect in classical conditioning.

The partial reinforcement extinction effect (PREE) refers to the phenomenon that conditioned responding extinguishes more slowly if subjects had been inconsistently ("partially") reinforced than if they had been reinforced on every trial ("continuously" reinforced). One largely successful account of the PREE, known as sequential theory (Capaldi, 1966), suggests that, when subjects are partially reinforced, they learn that memories of sequences of nonreinforced trials are associated with subsequent reinforcement. This association helps to maintain responding (i.e., delay extinction) when the subjects experience nonreinforced trials during extinction. Sequential theory's explanation of the PREE hinges on subjects learning sequences of nonreinforced trials during acquisition. However, direct evidence for such sequential learning is not available in previous studies of the PREE where animals are trained with multiple sequences of different lengths that are randomly intermixed and, therefore, cannot anticipate whether a given trial will be reinforced during acquisition. The current study conducted two experiments that trained rats with a single fixed trial sequence to provide evidence of sequential learning during conditioning, and then observe its effect on the PREE. Under one condition the rats did learn about the fixed sequence but did not subsequently show a PREE, whereas other rats that did show a PREE had not learned the trial sequences during conditioning. Therefore, contrary to sequential theory's prediction, our result suggests that learning about the trial sequence is neither necessary nor sufficient for the PREE. We suggest that the PREE may instead depend on uncertainty about whether the conditioned stimulus will be reinforced. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Partial reinforcement extinction and omission effects in the elimination and recovery of discriminated operant behavior.

Three experiments explored how training reinforcement schedules and context influence the elimination and recovery of human operant behavior. In Experiment 1, participants learned a discriminated operant response in Context A before the response was eliminated with extinction in Context B. They then received a final test in each context. Groups were trained with a discriminative stimulus that predicted a reinforced response on either every trial (continuous reinforcement [CRF]) or some of the trials (partial reinforcement [PRF]). Extinction was slower following PRF training (a partial reinforcement extinction effect [PREE]) and extinguished responding increased when tested in Context A ("ABA" renewal). Experiment 2 further confirmed the PREE was obtained equally whether extinction occurred in the training context (Context A) or a new context (Context B) which is consistent with trial-based accounts of the PREE. Experiment 3 used the same design as Experiment 1 to evaluate the influence of training reinforcement on response elimination with an omission contingency. Across the omission training phase in Context B, the decrease in responding occurred more slowly in the PRF-trained group in comparison to the CRF-trained group, perhaps the first demonstration of what might be termed a PRF omission effect. Again, ABA renewal was observed in Context A. Training reinforcement schedule therefore had a similar influence on response elimination with extinction and omission. Elimination and recovery of human instrumental behavior, with extinction or omission, are influenced by training reinforcement schedule and context. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Instrumental learning and behavioral persistence in children with attention-deficit/hyperactivity-disorder: does reinforcement frequency matter?

Prominent theoretical accounts of attention-deficit/hyperactivity-disorder (ADHD) hypothesize that reinforcement learning deficits underlie symptoms of ADHD. The Dynamic Developmental Theory and the Dopamine Transfer Deficit hypothesis assume impairments in both the acquisition and extinction of behavior, especially when learning occurs under partial (non-continuous) reinforcement, and subsequently the Partial Reinforcement Extinction Effect (PREE). Few studies have evaluated instrumental learning in ADHD and the results are inconsistent. The current study investigates instrumental learning under partial and continuous reinforcement schedules and subsequent behavioral persistence when reinforcement is withheld (extinction) in children with and without ADHD. Large well-defined samples of children with ADHD (n = 93) and typically developing (TD) children (n = 73) completed a simple instrumental learning task. The children completed acquisition under continuous (100%) or partial (20%) reinforcement, followed by a 4-min extinction phase. Two-way (diagnosis by condition) ANOVAs evaluated responses needed to reach the learning criterion during acquisition, and target and total responses during extinction. Children with ADHD required more trials to reach criterion compared to TD children under both continuous and partial reinforcement. After partial reinforcement, children with ADHD executed fewer target responses during extinction than TD children. Children with ADHD executed more responses than TD children during extinction, irrespective of learning condition. The findings demonstrate general difficulties in instrumental learning in ADHD, that is, slower learning irrespective of reinforcement schedule. They also show faster extinction following learning under partial reinforcement in those with ADHD, that is, a diminished PREE. Children with ADHD executed more responses during extinction. Results are theoretically important, with clinical implications for understanding and managing learning difficulties in those with ADHD, as they suggest poorer reinforcement learning and lower behavioral persistence.

Supplemental Material for Partial Reinforcement Extinction and Omission Effects in the Elimination and Recovery of Discriminated Operant Behavior

Supplemental Material for Partial Reinforcement Extinction and Omission Effects in the Elimination and Recovery of Discriminated Operant Behavior

Delaying extinction weakens the partial reinforcement extinction effect.

Conditioned responding that has been extinguished can spontaneously return when the conditioned stimulus (CS) is first presented after an extended delay. This spontaneous recovery of responding suggests that the memory of nonreinforced experience with the CS is impaired over the delay period. Rescorla (2007) provided evidence that this effect of time on nonreinforcement is not specific to extinction. He showed that a delay period can also reverse the reduction of responding established by a partial reinforcement schedule. Here we describe a series of experiments that attempted to confirm Rescorla's finding and additionally assessed the impact of the delay on another well-known consequence of partial reinforcement-the partial reinforcement extinction effect (PREE). Like Rescorla, we used a Pavlovian conditioning procedure with rats, measuring magazine activity during a CS that signaled food. Unlike Rescorla, we did not find that responding acquired under partial reinforcement spontaneously increased after a delay; however, we did observe a significant reduction in the PREE after that delay. We conclude that the passage of time has a selective effect on the retrieval of memories of nonreinforcement. Therefore, time produces spontaneous recovery by impairing retrieval of extinction memories but also weakens the PREE by impairing retrieval of memories of nonreinforcement that were acquired during partial reinforcement. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Partial reinforcement effects on acquisition and extinction of a conditioned taste aversion.

Four experiments with rat subjects asked whether a partial reinforcement extinction effect (PREE) occurs in taste aversion learning. The question has received little attention in the literature, and to our knowledge no taste aversion experiment has previously demonstrated a PREE. In each of the present experiments, experimental groups received a taste mixed in drinking water for 20 min; such taste exposures were sometimes paired with a lithium chloride (LiCl) injection and sometimes not. Control groups received only taste-LiCl pairings. There was evidence that each reinforced and non-reinforced trial produced increments and decrements in aversion strength (respectively), and trials mattered more than accumulated time during the conditioned stimulus and during the background (as emphasized in time-accumulation models like those of Gallistel and Gibbon, Psychological Review, 107, 289-344, 2000, and Gibbon and Balsam, Autoshaping and conditioning theory, Academic Press, New York, pp. 219-235, 1981). In addition, a partial reinforcement extinction effect was observed when there was a relatively large number of conditioning trials. The results extend our understanding of extinction in taste aversion learning and provide more evidence that aversion learning might follow rules that are qualitatively similar to those of other forms of learning.

The effect of partial and continuous reinforcement on the generalization of conditioned fear in humans

The popular paradigm of Pavlovian conditioned fear has been used to study learning and memory processes that mediate anxiety disorder. Fear extinction studies have often paired the conditioned stimulus (CS) and the unconditioned stimulus (US) on a subset of acquisition trials (partial reinforcement) to prolong extinction. However, there have been few studies on the effect of partial reinforcement and continuous reinforcement on fear generalization. To better understand the influence of the CS–US pairing rate on fear generalization, the experimenters used CSs paired with US at 50% (CS50), 75% (CS75), and 100% (CS100) during fear conditioning. A total of 66 participants, who were assigned at random to the different reinforcement rates, underwent fear acquisition and generalization tests on expectancy ratings and skin conductance responses. The continuous reinforcement group (CS100) showed a flatter generalization gradient, while the partial reinforcement groups (CS50, CS75) showed increased generalization magnitudes. These results suggest that the partial reinforcement extinction effect is related to threat intensity, whereas continuous reinforcement produces threat-sensitivity fear generalization, which has been used to explain the overgeneralization of fear.

Tribute to E. J. Capaldi: Celebration of a Psychological Scientist

Abstract E. J. (John) Capaldi (1928–2020) made numerous contributions to experimental psychology in his long career at the University of Texas at Austin and Purdue University. He was a pioneer in the area of animal learning and cognition, known for his sequential theory of partial reinforcement extinction effects. His research in this area and in memory and counting phenomena was conducted for the most part with rats running in straight alley mazes under various sequences of trial outcomes (e.g., reward, nonreward, variations in reward size). John's other interests included the sequential theory applied to Pavlovian conditioning, evolutionary theory, and psychology of science. More generally, John was a scientist to the core who served as a role model to those who knew him. His career centered on the values of science and rigorous critical thinking based in empirical data, coupled with a curiosity and openness to different and new ideas in psychology.

Individual differences in proactive interference in rats (Rattus Norvegicus)

Individual differences in behaviors are seen across many species, and investigations have focused on traits linked to aggression, risk taking, emotionality, coping styles, and differences in cognitive systems. The current study investigated whether there were individual differences in proactive interference tasks in rats (Rattus Norvegicus), and tested hypotheses suggesting that these tasks should load onto a single factor and there should be clusters of rats who perform well or poorly on these tasks. The performance of 39 rats was tested across three learning tasks that all involved disengagement from an irrelevant previously learned stimulus to a relevant stimulus: latent inhibition (LI), partial reinforcement extinction effect (PREE), and reversal learning (RL). An exploratory factor analysis revealed the existence of one factor underlying performance. A cluster analysis revealed the existence of sets of rats displaying either weak LI and strong PREE and RL effects, or vice versa. These findings suggest that proactive interference may be based on a single underlying psychological system in rats.

Effects of partial reinforcement on autoshaping in inbred Roman high- and low-avoidance rats

Individuals trained under partial reinforcement (PR) typically show a greater resistance to extinction than individuals exposed to continuous reinforcement (CR). This phenomenon is referred to as the PR extinction effect (PREE) and is interpreted as a consequence of uncertainty-induced frustration counterconditioning. In this study, we assessed the effects of PR and CR in acquisition and extinction in two strains of rats, the inbred Roman high- and low-avoidance (RHA and RLA, respectively) rats. These two strains mainly differ in the expression of anxiety, the RLA rats showing more anxiety-related behaviors (hence, more sensitive to frustration) than the RHA rats. At a neurobiological level, mild stress is known to elevate corticosterone in RLA rats and dopamine in RHA rats. We tested four groups of rats (RHA/CR, RHA/PR, RLA/CR, and RLA/PR) in two successive acquisition-extinction phases to try to consolidate the behavioral effects. Animals received training in a Pavlovian autoshaping procedure with retractable levers as the conditioned stimulus, food pellets as the unconditioned stimulus, and lever presses as the conditioned response. In Phase 1, we observed a PREE in lever pressing in both strains, but this effect was larger and longer lasting in RHA/PR than in RLA/PR rats. In Phase 2, reacquisition was fast and the PREE persisted in both strains, although the two PR groups no longer differed in lever pressing. The results are discussed in terms of frustration theory and of uncertainty-induced sensitization of dopaminergic neurons.

The partial reinforcement extinction effect depends on learning about nonreinforced trials rather than reinforcement rate.

Conditioned responding extinguishes more slowly after partial (inconsistent) reinforcement than after consistent reinforcement. This Partial Reinforcement Extinction Effect (PREE) is usually attributed to learning about nonreinforcement during the partial schedule. An alternative explanation attributes it to any difference in the rate of reinforcement, arguing that animals can detect the change to nonreinforcement more quickly after a denser schedule than a leaner schedule. Experiments 1a and 1b compared extinction of magazine responding to a conditioned stimulus (CS) reinforced with 1 food pellet per trial and a CS reinforced with 2 pellets per trial. Despite the difference in reinforcement rate, there was no reliable difference in extinction. Both experiments did demonstrate the conventional PREE comparing a partial CS (50% reinforced) with a consistent CS. Experiments 2 and 3 tested whether the PREE depends specifically on learning about nonreinforced trials during partial reinforcement. Rats were trained with 2 CS configurations, A and AX. One was partially reinforced, the other consistently reinforced. When AX was partial and A consistent, responding to AX extinguished more slowly than to A. When AX was consistent and A was partial, there was no difference in their extinction. Therefore, pairing X with partial reinforcement allowed rats to show a PREE to AX that did not generalize to A. Pairing A with partial reinforcement meant that rats showed a PREE to A that generalized to AX. Thus, the PREE depends on learning about nonreinforced trials during partial reinforcement and is not because of any difference in per-trial probability of reinforcement. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

A flexible and generalizable model of online latent-state learning.

Many models of classical conditioning fail to describe important phenomena, notably the rapid return of fear after extinction. To address this shortfall, evidence converged on the idea that learning agents rely on latent-state inferences, i.e. an ability to index disparate associations from cues to rewards (or penalties) and infer which index (i.e. latent state) is presently active. Our goal was to develop a model of latent-state inferences that uses latent states to predict rewards from cues efficiently and that can describe behavior in a diverse set of experiments. The resulting model combines a Rescorla-Wagner rule, for which updates to associations are proportional to prediction error, with an approximate Bayesian rule, for which beliefs in latent states are proportional to prior beliefs and an approximate likelihood based on current associations. In simulation, we demonstrate the model’s ability to reproduce learning effects both famously explained and not explained by the Rescorla-Wagner model, including rapid return of fear after extinction, the Hall-Pearce effect, partial reinforcement extinction effect, backwards blocking, and memory modification. Lastly, we derive our model as an online algorithm to maximum likelihood estimation, demonstrating it is an efficient approach to outcome prediction. Establishing such a framework is a key step towards quantifying normative and pathological ranges of latent-state inferences in various contexts.

Reinforcement Contingency Learning in Children with ADHD: Back to the Basics of Behavior Therapy

Reinforcement deficits in ADHD may affect basic operant learning processes relevant for Behavioral Treatment. Behavior acquired under partial reinforcement extinguishes less readily after the discontinuation of reinforcement than behavior acquired under continuous reinforcement, a phenomenon known as the Partial Reinforcement Extinction Effect [PREE], which has great relevance for the emergence of behavioral persistence. The present study examined acquisition and extinction of operant responding under partial and continuous reinforcement in children with and without ADHD. In addition, we evaluated the effectiveness of gradual stretching the reinforcement rate during acquisition for remedying potential acquisition or extinction deficits under partial reinforcement in ADHD. In an operant learning task designed to mimic the task confronted by an animal in a Skinner box, 62 typically developing and 49 children with ADHD (age: 8–12) were presented with a continuous, partial or gradually stretching reinforcement scheme followed by extinction. Both groups of children acquired the instrumental response more slowly and exhibited more behavioral persistence (reduced extinction) when responding was initially reinforced under partial relative to continuous reinforcement, with no differences between groups. Progressive ratio stretching resulted in faster acquisition than partial reinforcement yet promoted equal behavioral persistence, again without differences between ADHD and TD groups. Unlike suggested by previous research, children with ADHD exhibit neither an acquisition deficit under partial reinforcement nor a deficit in PREE. Of relevance for Behavioral Treatment, gradual reinforcement stretching can be used to facilitate response acquisition over purely partial reinforcement while maintaining equal behavioral persistence upon reward discontinuation.

The partial-reinforcement extinction effect does not result from reduced sensitivity to nonreinforcement.

Five experiments used a magazine approach paradigm with rats to investigate whether learning about nonreinforcement is impaired in the presence of a conditioned stimulus (CS) that had been partially reinforced (PRf). Experiment 1 trained rats with a PRf CS and a continuously reinforced (CRf) CS, then extinguished responding to both CSs presented together as a compound. Probe trials of each CS presented alone revealed that extinction was slower for the PRf CS than the CRf CS, despite being extinguished in compound. In Experiment 2, a CRf light was extinguished in compound with either a CRf CS or a PRf CS that had been matched for overall reinforcement rate. Responding to the light extinguished at the same rate regardless of the reinforcement schedule of the other CS. Experiment 3 replicated this result with a PRf light. Thus, we found no evidence that a PRf CS impairs extinction of another CS presented at the same time. Experiments 4 and 5 extended this approach to study the acquisition of conditioned inhibition by training an inhibitor in compound with either a PRf or CRf excitatory CS. The reinforcement schedule of the excitatory CS had no effect on the acquisition of inhibition. In sum, conditioning with a PRf schedule slows subsequent extinction of that CS but does not affect learning about the nonreinforcement of other stimuli presented at the same time. We conclude that the Partial Reinforcement Extinction Effect is not attributable to a decrease in sensitivity to nonreinforcement following presentation of a PRf CS. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

The partial reinforcement extinction effect: The proportion of trials reinforced during conditioning predicts the number of trials to extinction.

Four experiments compared the extinction of responding to a continuously reinforced (CRf) conditioned stimulus (conditional stimulus [CS]) consistently reinforced on every trial, with extinction of responding to a partially reinforced (PRf) CS that had been inconsistently reinforced. To equate the acquisition of responding between the two CSs, the average duration of the CRf CS was extended so that it scheduled the same overall rate of reinforcement per unit time as the PRf CS. Experiment 1 used a within-subjects design to compare the rates of extinction for a 10-s PRf CS reinforced on 33% of trials versus a 30-s CRf CS. Experiment 2 made the same comparison but using a between-subjects design. Experiment 3 compared extinction in a group trained with a 10-s PRf CS reinforced on 20% of trials and a group trained with a 50-s CRf CS. Experiment 4 compared the rates of extinction following two partial reinforcement schedules: a 10-s PRf CS reinforced on 33% of trial versus a 20-s CRf CS reinforced on 66% of trials. In each experiment, responding took longer to extinguish for the CS that scheduled a lower per-trial probability of reinforcement. Modeling of individual extinction curves using Weibull functions indicated that the latency to initiate extinction was directly related to the per-trial probability of reinforcement learned during acquisition. For example, compared with training with a CRf CS, rats reinforced on 33% of trials took approximately 3 times as many trials to initiate extinction, and rats reinforced on 20% of trials took 5 times as many trials to initiate extinction. These results provide support for trial-based accounts of extinction (e.g., Capaldi & Deutsch, 1967), whereby rats learn about the expected number of trials per reinforcer, and extinction depends on the number of expected reinforcers that have been omitted rather than on the number of extinction trials per se. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Working Memory and Reinforcement Schedule Jointly Determine Reinforcement Learning in Children: Potential Implications for Behavioral Parent Training.

Introduction: Behavioral Parent Training (BPT) is often provided for childhood psychiatric disorders. These disorders have been shown to be associated with working memory impairments. BPT is based on operant learning principles, yet how operant principles shape behavior (through the partial reinforcement (PRF) extinction effect, i.e., greater resistance to extinction that is created when behavior is reinforced partially rather than continuously) and the potential role of working memory therein is scarcely studied in children. This study explored the PRF extinction effect and the role of working memory therein using experimental tasks in typically developing children.Methods: Ninety-seven children (age 6–10) completed a working memory task and an operant learning task, in which children acquired a response-sequence rule under either continuous or PRF (120 trials), followed by an extinction phase (80 trials). Data of 88 children were used for analysis.Results: The PRF extinction effect was confirmed: We observed slower acquisition and extinction in the PRF condition as compared to the continuous reinforcement (CRF) condition. Working memory was negatively related to acquisition but not extinction performance.Conclusion: Both reinforcement contingencies and working memory relate to acquisition performance. Potential implications for BPT are that decreasing working memory load may enhance the chance of optimally learning through reinforcement.

Partial reinforcement and conditioned taste aversion: No evidence for resistance to extinction.

The partial reinforcement extinction effect (PREE) is the observation that, following training in which a response is followed by reward on only a subset of trials, the response is more resistant to extinction following the total removal of reward than it is after training in which reward is presented on all trials. The PREE is almost ubiquitous in instrumental conditioning procedures but only inconsistently observed in Pavlovian conditioning. In his classic review of animal learning, Mackintosh attributes the bulk of the PREE to generalisation decrement relating to the fact that partial reinforcement typically ensures that acquisition of responding has taken place in conditions similar to that of extinction (e.g., in the absence of the reinforcer). We report here that extinction of a conditioned taste aversion is not retarded by partial reinforcement in terms of either consumption of the taste or hedonic reactions to it (assessed through the analysis of licking microstructure). These results are consistent with Mackintosh's analysis of the PREE and the way in which it might differ between instrumental and Pavlovian conditioning.

Bayesian methods for addressing long-standing problems in associative learning: The case of PREE.

Most associative models typically assume that learning can be understood as a gradual change in associative strength that captures the situation into one single parameter, or representational state. We will call this view single-state learning. However, there is ample evidence showing that under many circumstances different relationships that share features can be learned independently, and animals can quickly switch between expressing one or another. We will call this multiple-state learning. Theoretically, it is understudied because it needs a different data analysis approach from those usually employed. In this article, we present a Bayesian model of the Partial Reinforcement Extinction Effect (PREE) that can test the predictions of the multiple-state view. This implies estimating the moment of change in the responses (from the acquisition to the extinction performance), both at the individual and group levels. We used this model to analyze data from a PREE experiment with three levels of reinforcement during acquisition (100%, 75% and 50%). We found differences in the estimated moment of switch between states during extinction, so that it was delayed after leaner partial reinforcement schedules. The finding is compatible with the multiple-state view. It is the first time, to our knowledge, that the predictions from the multiple-state view are tested directly. The article also aims to show the benefits that Bayesian methods can bring to the associative learning field.

Affective\u2013associative two-process theory: a neurocomputational account of partial reinforcement extinction effects

The partial reinforcement extinction effect (PREE) is an experimentally established phenomenon: behavioural response to a given stimulus is more persistent when previously inconsistently rewarded than when consistently rewarded. This phenomenon is, however, controversial in animal/human learning theory. Contradictory findings exist regarding when the PREE occurs. One body of research has found a within-subjects PREE, while another has found a within-subjects reversed PREE (RPREE). These opposing findings constitute what is considered the most important problem of PREE for theoreticians to explain. Here, we provide a neurocomputational account of the PREE, which helps to reconcile these seemingly contradictory findings of within-subjects experimental conditions. The performance of our model demonstrates how omission expectancy, learned according to low probability reward, comes to control response choice following discontinuation of reward presentation (extinction). We find that a PREE will occur when multiple responses become controlled by omission expectation in extinction, but not when only one omission-mediated response is available. Our model exploits the affective states of reward acquisition and reward omission expectancy in order to differentially classify stimuli and differentially mediate response choice. We demonstrate that stimulus–response (retrospective) and stimulus–expectation–response (prospective) routes are required to provide a necessary and sufficient explanation of the PREE versus RPREE data and that Omission representation is key for explaining the nonlinear nature of extinction data.